Specification Mutation for Test Generation and Analysis

Mutation analysis is a fault-based testing technique that uses mutation operators to introduce small changes into a program or specification, producing mutants, and then chooses test cases to distinguish the mutants from the original. Mutation operators differ in the coverage they get. They also differ in the number of mutants they generate. Consequently, selecting mutation operators is an important problem whose solution affects the effectiveness and cost of mutation testing. We use the automated test generation and evaluation method that combines a model checker and mutation analysis. We define a set of mutation operators and implement a mutation gener­ ator for specifications written in SMV, a popular model checker. To select the most effective mutation operators and sets of operators, we compare them using both theoretical and experimental methods. We construct mutation detection conditions and develop a technique to theoretically compare mutation operators. We apply mutation coverage and pairwise coverage metrics to empirically compare the effectiveness of mutation operators. To detect a fault in a program, a test case must cause the fault to affect the outputs, not just intermediate variables. We develop a method that uses a model checker to guarantee that tests cause visible output failures. We find that mutation operators form a hierarchy with respect to detection capability; we can skip a test for a mutation from an easier-to-detect mutation operator in the hierarchy, provided that we detect a corresponding mutation from a harder-to-detect operator. Our theoretical technique allows us to prove that the hierarchy applies to arbitrary logic expressions, whereas previous results apply only to logic expressions in disjunctive normal form. Based on analysis and empirical evaluation, we recommend mutation operators and sets of mutation operators that yield good test coverage at a reduced cost. Our experiments show that specification-based mutation can be applied to test programs; it gets good program-based coverage. Our method for guaranteeing fault visibility is very effective for black-box testing of programs which have a large intermediate state. This thesis shows that specification-based mutation can be used to economically generate effective tests.